Design of store-and-forward servers for digital media distribution

Production of high quality digital media is increasing in both the commercial and academic world. This content needs to be distributed to end users on demand and efficiently. Initiatives like CineGrid [1] push the limit looking at the creation of content distribution centres connected through dedicated optical circuits. The research question of this project is the following: “What is the optimal architecture for the (CineGrid) storage systems that store and forward content files of a size of hundreds of GBs?” First I made an overview of the current situation. At the moment the Rembrandt cluster nodes [16] are used in the storage architecture. All data has to be transferred manually to the nodes via FTP. This is not preferred, because administration is difficult. Therefore a list of criteria is made for the new storage architecture. Important criteria are bandwidth (6.4 Gb/s) and space (31.2 TB a year and expandable). I made a comparison between open source distributed parallel file systems based on these criteria. Lustre and GlusterFS turned out to be the best of these file systems according to the criteria. After that I proposed two architectures which use these file systems. The first architecture contains only cluster nodes and the second architecture contains cluster nodes and a SAN. In the end it is recommended to install GlusterFS in the first architecture on the existing DAS3 nodes [15] with Ethernet as interconnect network. This way costs can be saved for a SAN (considering the limited budget). Further tests are necessary to determine the exact configuration.